Augmented reality apparatus

ABSTRACT

An embodiment of the invention relates to an apparatus and related methods for providing a person with an augmented reality, the apparatus comprising: a projector that projects light that generates an augmenting visual image (AVI); optics that directs to a camera a portion of the projected light and a portion of light arriving from a scene so that a combined image comprising an image of the scene and the AVI are generated in a photosensor in the camera; and a controller that processes the combined image to compare positions of homologous features in the image of the scene with positions of corresponding location markers in the AVI, and renders an adjusted AVI based on the comparison, so that the location markers and the corresponding homologous features are substantially coincident in the combined image.

RELATED APPLICATIONS

The present application claims the benefit under 35 U.S.C. 119(e) ofU.S. Provisional Application 61/809,415 filed on 7 Apr. 2013, thedisclosure of which is incorporated herein by reference in its entirety.

FIELD

Embodiments of the invention relate to an apparatus for providing aperson with an augmented reality.

BACKGROUND

In Augmented Reality (“AR”) technology, natural vision is supplementedwith the addition of an augmenting visual image (“AVI”) that overlays aportion of a real time scene in a region of interest in the person'snatural field of view (“FOV”). Hereinafter, the portion of the real timescene may be referred to as a “scene”. The AVI is typicallycomputer-generated, and is rendered from a computer code (also referredto as an “AVI model”) that is stored in a computer memory. The AVI maycomprise one or more features (“AVI constructs”) that are viewed by theperson together with the scene. The AVI constructs may, for example,comprise an image of an object or region in the scene, or an imagepresenting information regarding the object or region in the scene. Aportion of the AVI model that encodes the AVI construct may be referredto as a “construct model”.

Typically, the AVI is responsive to the scene, and changes as the scenechanges. For example, AVI constructs that are associated with aparticular feature in a scene may appear to the person, when viewedtogether with the scene, to be co-localized with the associated feature.As the person's FOV shifts, the AVI may change and/or shift accordinglyto maintain contextual and spatial relevance with the scene.

SUMMARY

An aspect of an embodiment of the invention relates to an apparatus,also referred to as a “referenced augmented reality” (“RAR”) apparatusthat renders an AVI from an AVI model and combines it with an image of ascene (“scene image”) that a person is looking at on the person's retinaso that the person sees a combined image in which the AVI is relativelyaccurately aligned with the scene image. Hereinafter the combined imagein accordance with an embodiment of the invention may be referred to asa “RAR image”. According to an embodiment of the invention, the AVIincludes at least one AVI construct and a plurality of location markersthat correspond to locations of selected homologous features in thescene. Accordingly, the AVI model comprises at least one construct modelthat encodes the at least one AVI construct as well as a “marker model”,which is a computer code that encodes the location markers. The RARapparatus captures and analyzes the RAR image to compare positions ofthe homologous features in the scene image with positions of thelocation makers in the AVI, and based on the comparison, if necessary,adjusts the AVI so that the location markers and the correspondinghomologous features are substantially coincident in the RAR image.

For convenience of presentation, the homologous features in the scenemay be referred to as “fiducials”, and the location markers in the AVImay be referred to as “AVI markers”.

According to an embodiment of the invention, the RAR apparatus comprisesa controller that has the AVI model stored in its computer memory. Thecontroller renders the AVI in a projector and controls the projector toproject light that forms the AVI (hereinafter referred to as “AVIlight”). The RAR further comprises optics that directs a portion oflight arriving from the scene that the person is looking at (hereinafterreferred to as “scene light”) and a portion of the AVI light to theperson so that the person sees the RAR image comprising the scene imageoverlaid with the AVI. The RAR apparatus further comprises a camera. Theoptics directs portions of the scene light and the AVI light to thecamera to generate a copy of the RAR image on the photosensor of thecamera.

In an embodiment of the invention, the controller is operable to processthe copy of the RAR image captured by the camera to compare positions ofthe fiducials in the scene image with the positions of the homologousAVI markers in the AVI and render an adjusted AVI based on thecomparison, so that the AVI markers and the images of the fiducials aresubstantially coincident in the RAR image.

The RAR image directed to the person may be referred to as a “person RARimage” or “PRAR image” and the copy of the RAR image directed to thecamera may be referred to as a “camera RAR image” or “CRAR image”. ThePRAR image and the CRAR image may be referred to generically as the RARimage or combined image. In accordance with an embodiment of theinvention, the PRAR image and the CRAR image are “substantiallydimensionally identical”, that is, except for a possible linear scalefactor, the RAR apparatus is substantially free of sources of relativedimensional variances between the PRAR image and the CRAR image. Thesources of relative dimensional variances may include parallax error dueto the positioning of the eye of the person and the camera, differentialchromatic magnification and/or optical aberrations.

According to an embodiment of the invention, the optics may include apartially transmitting/partially reflecting mirror, which receives thescene light and the AVI light, and splits them so that portions of thescene light and the AVI light are directed to the person to generate thePRAR image and portions of the scene light and the AVI light aredirected to the camera to generate the CRAR image. In an embodiment ofthe invention, the optics may optionally be configured to direct thescene light and the AVI light to the person to generate a PRAR image onthe retina of a user. The optics is further configured to receive areflection of the PRAR image from the retina, and direct the reflectionof the PRAR image to the camera to generate the CRAR image.

An aspect of an embodiment of the invention relates to a method forproviding a person with an AR responsive to a scene that the personviews, the method comprising: rendering an AVI that has AVI markershomologous to fiducials in the scene; projecting the AVI; directing theAVI and an image of the scene to the person's eye and a camera;capturing the AVI and the image of the scene; comparing positions of AVImarkers in the captured AVI with positions of homologous fiducials inthe captured scene image; rendering an adjusted AVI in which the AVImarkers are substantially coincident with the homologous fiducials inthe capture scene image; projecting the adjusted AVI; and directing theadjusted AVI and the image of the scene to the person's eye and thecamera.

In the discussion, unless otherwise stated, adjectives such as“substantially”, “relatively” and “about” modifying a condition orrelationship characteristic of a feature or features of an embodiment ofthe invention, are understood to mean that the condition orcharacteristic is defined to within tolerances that are acceptable foroperation of the embodiment for an application for which it is intended.Unless otherwise indicated, the word “or” in the specification andclaims is considered to be the inclusive “or” rather than the exclusiveor, and indicates at least one of, or any combination of items itconjoins.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF FIGURES

Non-limiting examples of embodiments of the invention are describedbelow with reference to figures attached hereto that are listedfollowing this paragraph. Identical structures, elements or parts thatappear in more than one figure are generally labeled with a same numeralin all the figures in which they appear. Dimensions of components andfeatures shown in the figures are chosen for convenience and clarity ofpresentation and are not necessarily shown to scale.

FIG. 1 schematically illustrates a RAR apparatus, in accordance with anembodiment of the invention;

FIGS. 2A-2C schematically illustrates a scene image, an AVI and a RARimage that is a combination of the scene image and the AVI, inaccordance with an embodiment of the invention;

FIG. 3A schematically illustrates a scene viewed from two points ofview;

FIGS. 3B-3C schematically illustrates a scene image, an AVI and a RARimage, corresponding to the two points of view of FIG. 3A, in accordancewith an embodiment of the invention;

FIG. 4A schematically illustrates a split-field RAR apparatus, inaccordance with an embodiment of the invention;

FIG. 4B schematically illustrates a split-field RAR image having anoccluded field overlaid with a replacement AVI construct, in accordancewith an embodiment of the invention.

FIG. 5 schematically illustrates a surgery-related example of a scene, ascene image, an AVI and a RAR image, in accordance with an embodiment ofthe invention;

FIGS. 6A-6B schematically illustrates a RAR apparatus and optical pathstaken by light from various visual inputs in the optics of the RARapparatus, in accordance with an embodiment of the invention;

FIGS. 7A-7C schematically illustrates another RAR apparatus and opticalpaths taken by light from various visual inputs in the optics of the RARapparatus, in accordance with an embodiment of the invention; and

FIGS. 8A-8B shows flowcharts for a method for providing a person with anAR responsive to a scene that the person views, in accordance with anembodiment of the invention.

DETAILED DESCRIPTION

In the following detailed description, the components of a RAR apparatusin accordance with an embodiment of the invention are schematicallyillustrated in FIG. 1 and discussed with reference to that figure. FIGS.2A-2C and 3A-3C and the related descriptions describe providing a sceneimage, an AVI, and a RAR image for a scene having an object embeddedwithin a substrate. The components of a split-field RAR apparatus and anexample of a split-field RAR image are schematically illustrated inFIGS. 4A-4B and discussed with reference to those figures. FIG. 5 andthe related description describe providing a scene image, an AVI, and aRAR image for a scene comprising an area around a kidney of a patient insurgery. It will be appreciated that the images (scene image, AVI andRAR images), as generated on a retina of a person or a photosensor of acamera, are inverted (up-down) and reversed (right-left). For clarity ofpresentation, the images are presented in the figures as right-side-upand not reversed (right-left). Further examples of a RAR apparatus inaccordance with an embodiment of the invention, with various exemplaryoptics and its details and functions, are discussed with reference toFIGS. 6A-6B and 7A-7C. Flowcharts of a method for providing an ARresponsive to a scene that the person views, in accordance with anembodiment of the invention, are shown in FIGS. 8A-8B and discussed withreference to the figures.

Reference is now made to FIG. 1, which schematically shows a RARapparatus 100 for providing a person 60 with an AR responsive to a scenethat the person is looking at, schematically represented by an ellipse72, in a region of interest in a field of view 50 of the person. Scenelight that arrives from the scene to RAR apparatus 100 is schematicallyrepresented by a block arrow 72L. RAR apparatus 100 comprises optics110, a camera 120, a projector 130, and a controller 140 connected bywire or wirelessly with the projector and the camera. Controller 140 hasin its computer memory an AVI model (not shown) used to render, inprojector 130, an AVI 74 (schematically represented by an ellipse 74).Projector 130 projects AVI light that is schematically represented by ablock arrow 74L. The AVI model includes at least one construct model anda marker model that are used to generate, respectively, at least one AVIconstruct (not shown in FIG. 1) and a plurality of AVI markers (notshown in FIG. 1) as parts of AVI 74. The AVI markers are homologous tofiducials in scene 72 (not shown in FIG. 1). Schematic illustrations ofexemplary scene images with fiducials and exemplary AVIs with AVImarkers, as well as exemplary RAR images, are shown in FIGS. 2A-2B andfigures following FIGS. 2A-2B and are discussed with reference to thefigures below.

Optics 110 directs a portion of scene light 72L and a portion of AVIlight 74L that it receives to person 60 so that person 60 sees a PRARimage, schematically represented by an ellipse 76, that comprises theimage of scene 72 overlaid by AVI 74. The combination of the portion ofscene light 72L and the portion of AVI light 74L directed by optics 110to person 60 to generate PRAR image 76 is referred to as “PRAR light”.The PRAR light is schematically represented by a block arrow 76L. Optics110 directs a portion of scene light 72L and a portion of AVI light 74Lto a photosensor (not shown) of camera 130 to generate a CRAR image,schematically represented by an ellipse 77, that comprises the image ofscene 72 overlaid by AVI 74. The combination of the portion of scenelight 72L and the portion of AVI light 74L directed by optics 110 to thephotosensor of camera 130 to generate CRAR image 77 is referred to as“CRAR light”. The CRAR light is schematically represented by a blockarrow 77L.

In accordance with an embodiment of the invention, as discussed below,controller 140 is operable to process CRAR image 77 to compare thepositions of the fiducials with the positions of the homologous AVImarkers and render an adjusted AVI 74 based on the comparison forprojector 130 to project, so that the AVI markers and the fiducials aresubstantially coincident in PRAR image 76. In accordance with anembodiment of the invention, the PRAR image and the CRAR image aresubstantially dimensionally identical. In certain embodiments of theinvention, dimensional variations between the PRAR image and the CRARimage, if present, may be digitally compensated for by controller 140.For example the differences in lens focal length for differentwavelength light may be accounted for and compensated. In certainembodiments of the invention, there may be differences between theintensity of light forming PRAR image 76 and the intensity of lightforming CRAR image 77.

Reference is now made to FIG. 2A which schematically shows, by way ofexample, a scene image 73 of a possible scene 72 (of FIG. 1), AVI 74rendered by RAR apparatus 100 (of FIG. 1), and a RAR image 75 (that maybe PRAR 76 or CRAR 77 of FIG. 1), which is a combination of scene image73 and AVI 74, in accordance with an embodiment of the invention.

In FIG. 2A, by way of example, scene image 73 comprises an object 84partially embedded within an opaque substrate 82, so that only a portionof object 84 (shown with a continuous line) protruding from substrate 82is visible in scene image 73. A remainder of object 84 (shown with adashed line) that is inside substrate 82 is not visible in scene image73. Further by way of example, substrate 82 has three features indicatedby solid circles 88A-88C which may be suitable to function as fiducials.

AVI 74 comprises an AVI construct 86 which by way of example is arepresentation of object 84 in its entirety, showing the embeddedportion of object 84 in addition to the visible portion of the object.AVI 74 further includes AVI markers 89A-89C that are homologous tofiducials 88A-88C.

RAR image 75, which is a combination of scene image 73 and AVI 74,comprise AVI construct 86 coincident with object 84, as well asfiducials 88A-88C coincident with AVI markers 89A-89C.

Object 84 and substrate 82 may be in any shape, and they may be in anyspatial arrangement. The identity of the object and the substrate may bedifferent depending on the particular application of the RAR apparatus.For example, substrate 82 may be a wood block, and object 84 may be anail embedded within it. Alternatively, object 84 may be a plug that isplaced inside an electronic device represented as substrate 82.Alternatively, object 84 may be an airplane that is entering a cloudrepresented by substrate 82. In a surgical setting, substrate 82 may bea bone segment of a patient and object 84 may be a setting pin.Alternatively, substrate 82 may be a portion of a patient and object 84may be an organ, a blood vessel, a tissue, a tumor, a surgicalinstrument, or the like. There may be multiple substrates and multipleobjects (which may or may not be embedded in a substrate), onlysubstrates, only objects, only one substrate or only one object. Otherexamples of the identity of the object and the substrate in the contextof various applications are described hereinbelow.

In certain embodiments of the invention, the AVI may be projected at aframe rate that allows changes in the AVI to be perceived by the personviewing the PRAR image to be a moving image. As such, the AVI may beprojected from the projector at a frame rate of between about 15 framesper second (FPS) to about 100 FPS.

According to an embodiment of the invention, the AVI construct in theAVI may be rendered from a construct model stored in a computer memoryof controller 140 (FIG. 1) of RAR apparatus 100, as a part of the AVImodel.

According to an embodiment of the invention, the construct model mayencode a two-dimensional representation or a three-dimensionalrepresentation of a corresponding object in the scene. In certainembodiments of the invention, the construct model may be based on aknown shape and dimensions of the corresponding object. The constructmodel may, for example, be generated from drawings, CAD files, maps,and/or another previously prepared representation of the correspondingobject. Alternatively, the construct model may be reconstructed from oneor more images of an equivalent object having the same size anddimensions. In certain embodiments of the invention, the construct modelmay be reconstructed from one or more images of the same object that wascaptured previously, for example, before the object was placed in thescene (or embedded in the substrate). Alternatively, the construct modelmay be reconstructed from one or more images of the object capturedbefore or during the viewing session with the RAR apparatus. In certainembodiments of the invention, the images may be optical images, oralternatively may be captured by means of an alternative non-opticalimaging modality, such as but not limited to X-ray, magnetic resonanceimaging (MRI), ultrasound or the like, that allows for capturing one ormore images of a portion of the object or the object in its entirety(including, for example, any portions that may be embedded within thesubstrate). The imaging modality through which the construct model isreconstructed may be a tomographic imaging modality such as but notlimited to a CT scan, a PET scan, a tomographic MRI, an ultrasoundtomography or the like. In certain embodiments of the invention, theconstruct model may be periodically updated, with imaging sessionsinterspersed between, or performed concurrently with, viewing with theRAR apparatus. Periodically updating the construct model may beadvantageous in cases where the shape or other features of the objectchanges over time, for example through manipulation of the object by aperson.

Alternatively or additionally, the AVI construct may be an added visualcontent that is not a representation of physical features of acorresponding object or region in the scene, but provides informationabout the object or region. The visual content may be, for example, apiece of text such as a symbol, a name, a description, or a parameter(for example, distance from person, speed, location coordinates, or thelike). Alternatively or in addition, the added visual content may be agraphic, for example an expected trajectory of the object, a plannedplacement or path of a non-existing object, a perimeter representing asafe distance from the object or region, a flashing circle highlightingthe object/region or the like. Alternatively or in addition, the addedvisual content may be a video stream, for example from a camera in aminimally invasive surgery tool. Many other added types of visualcontent may occur to a person skilled in the relevant art.

According to an embodiment of the invention, the AVI further includes aplurality of AVI markers. As shown in FIGS. 2A-2C, by way of example,AVI markers 89A-89C appear as targets, with the center of the fiducials88A-88C located at the intersection of the target cross-hairs when theAVI markers and the homologous fiducials are coincident in RAR image 75.The AVI markers are typically computer generated, and may be presentedas having any shape and/or color as specified, for example, as a redhollow circle, a yellow cross, or any other shape/color.

According to an embodiment of the invention, the AVI markers aspresented in the AVI may be rendered from a marker model stored in thecomputer memory of the controller of the RAR apparatus as a part of theAVI model. The marker model encodes at least the locations of thefiducials. When the AVI is rendered from the AVI model, appropriate AVImarkers are concomitantly rendered from the marker model as a part ofthe AVI. In certain embodiments of the invention, the marker model maybe based on known locations of fiducials in a scene. The marker modelmay, for example, be generated from drawings, CAD files, maps, and/oranother previously prepared representation of the scene. In certainembodiments of the invention, the marker model may be a reconstructionof the location of the fiducials based on one or more captured images ofthe scene. The captured images of the scene may optionally be opticalimages, or alternatively captured through an alternative imagingmodality, such as but not limited to X-ray, MRI, ultrasound and thelike. The imaging of the scene through which the marker model isreconstructed may be a three-dimensional tomographic (MRT) imaging suchas but not limited to a CT scan, a PET scan, a tomographic MRI, anultrasound tomography or the like. In certain embodiments of theinvention, the imaging of the scene may be performed prior to viewingwith the RAR apparatus. Alternatively, the marker model may beperiodically updated with imaging sessions interspersed between, orperformed concurrently with, viewings with the RAR apparatus.Periodically updating the marker model may be advantageous in caseswhere the locations of the fiducials relative to each other change overtime due to, for example, movement or manipulation of the fiducials orthe object.

In certain embodiments of the invention, the fiducials may optionally benative features naturally present in one or more items in the scene, orbe artificial indicators applied to various locations in or around thescene. Artificial indicators may be advantageous in cases where thescene lacks native features that are sufficiently visible or distinct toserve as fiducials, for example for positional analysis with AVI markersor for the purpose of generating a marker model. The artificialindicators may be applied in any form as appropriate, for example, as apin, a paint marking, a tattoo marking, an ink marking, a sticker, orthe like. Further, the artificial indicators may include a high contrastmaterial that has sufficient contrast or distinction to serve asfiducials. By way of example, artificial indicators for CT scanning mayhave materials that are highly opaque to X-ray, artificial indicatorsfor MRI may have materials that are highly responsive to RF-inducedmagnetic resonance, and the like.

By way of example as shown in FIGS. 2A-2C, fiducials 88A-88C may belocated on substrate 82. Alternatively or additionally, the fiducialsmay optionally be located on any feature, object or region in the scene.

According to an embodiment of the invention, as shown in FIG. 2A, thescene image comprises at least three fiducials 88A-88C. Typically, thefiducials are not all in one plane. More fiducials may be used forbetter accuracy. Alternatively, a curved line may be used as a fiducialhaving substantially the same utility as a plurality of points.

FIG. 2B shows a RAR image 75 after a shift in the scene and concomitantshift in scene image 73 and before the AVI is adjusted accordingly. Byway of example as shown in FIG. 2B, substrate 82, together with object84 is moved laterally to the right and upwards. Such a shift may becaused by a change in the position of the person, or by substrate 82(with object 84) being moved. As a result of the shift, in RAR image 75,AVI markers 89A-89C are no longer coincident with the homologousfiducials 89A-89C and AVI construct 86 is no longer coincident withobject 84. When such a shift occurs, the misalignment of AVI markers89A-89C and homologous fiducials 88A-88C in RAR image 75 is detected bythe RAR apparatus (in the case of RAR apparatus 100 shown in FIG. 1, bycontroller 140), which then renders an adjusted AVI 74. As shown in FIG.2C, RAR image 75 with the adjusted AVI includes adjusted positions ofAVI markers 89A-89C as well as a concomitantly adjusted position of AVIconstruct 86. As a result, the substantial coincidence of the AVImarkers with the fiducials, and of the AVI construct with thecorresponding object, is restored in RAR image 75. Similar adjustmentsof the AVI based on the positional comparison of the AVI markers withthe fiducials in the RAR image may be made with different types ofshifts in the scene, such as changes in the proximity of the object tothe person or the orientation of the object in relation to the person(rotation).

In certain embodiments of the invention, adjustment of the AVI may bebased on other factors in addition to the positional comparison of theAVI markers in the AVI with the fiducials in the scene image. Suchadditional factors may optionally include one or a combination of: anassumed position of the person relative to the object, data fromexternal sensors configured to detect the position of the person's head,a user-provided estimated head position, or the like.

According to an embodiment of the invention, the adjustment of AVI 74 isa repeating process. That is, after a RAR image is analyzed to comparethe alignment of the AVI markers with the homologous fiducials, anadjusted AVI is rendered based on the analysis and projected to theperson. The resulting adjusted RAR image may then be analyzed to rendera yet further adjusted AVI, and so on. This repeating process allows forthe alignment between the scene image and the AVI to be continuouslyimproved in an iterative process, in which the AVI is repeatedlyadjusted until the degree of alignment between the AVI markers and thefiducials in the RAR image reach or go below a predefined threshold.Additionally or alternatively, the repeating process allows for the AVIto be continuously responsive to changes in the scene. In certainembodiments of the invention, the repeating process may include creatingan updated construct model and/or an updated marker model.

It will be appreciated that the initial AVI at the beginning of aviewing session with the RAR apparatus is not rendered based on apositional comparison of AVI markers with the homologous fiducials inthe RAR image because there are no AVI markers yet presented. Theinitial rendering of the AVI may, thus, be determined based onestimation methods that do not require comparing the positions of theAVI markers with the positions of the fiducials in the RAR image. Theestimation methods may optionally be one or a combination of: an assumedposition of the person and the object, data from external sensorsconfigured to detect the position of the person's head, a user-providedestimated head position, or the like. Additionally or alternatively, theinitial AVI may be determined based on an analysis of the positions ofthe fiducials in the initially captured scene image without comparisonwith AVI markers. In certain embodiments of the invention, the initialAVI may comprise the AVI markers without images of AVI constructs, andthe AVI construct is presented following the adjusting of the AVI basedon the positional comparison of the AVI markers with the homologousfiducials in the RAR image.

In certain embodiments of the invention, the AVI markers may bepresented in a mode that is not visible to the person while beingdetectable by the camera. For the AVI to be adjusted by the controllerbased on the comparison of the fiducials and AVI markers, the AVImarkers have to be detected by the camera, but do not have to be visibleto the person. Therefore, the AVI markers may be projected, optionally,as a series of pulses having a duration that is too short to beperceived by a human, or at a wavelength that is detectable by thecamera but is not visible to the human eye such as in the infraredwavelength range or the ultra-violet wavelength range. This may serve anumber of uses, such as making the RAR image easier to interpret for theperson viewing it, or aiding in the processing by the controller of theRAR image by making the AVI markers readily separable from thefiducials. In certain embodiments of the invention, the fiducials may benot visible to the person while being detectable by the camera, forexample, infrared point sources that are at a wavelength that isdetectable by the camera but not visible to the person.

With reference to FIGS. 3A-3C, in certain embodiments of the invention,the construct model may be a three-dimensional representation of anobject in the scene, and the AVI construct may be a two-dimensionalrendering from the construct model presenting the object as seen from aselected point of view. FIG. 3A shows a perspective view of a possiblescene 172 having a substrate 182 with an embedded object 184, withillustrated arrows indicating point of view 1 (“POV1”) and point of view2 (“POV2”). By way of example, substrate 182 is shown with threefeatures indicated by solid balls 188A-188C, which are suitable tofunction as fiducials.

FIG. 3B illustrates a scene image 173, an AVI 174 and a RAR image 175based on the person viewing the scene 172 having substrate 182 andobject 184 from POV1. In scene image 173, substrate 182 is visibletogether with fiducials 188A-188C, while object 184, being obscured bysubstrate 182, is not visible. AVI 174 includes AVI markers 189A-189Cthat are homologous to fiducials 188A-188C and an AVI construct 186,which is a two-dimensional image representing object 184 as seen fromPOV1 that is rendered from a construct model that is a three-dimensionalrepresentation of object 184. AVI construct 186, presented to becoincident with the correct location of non-visible object 184 in RARimage 175, appears as a circle, corresponding to the orientation ofobject 184 with respect to POV1.

FIG. 3C illustrates scene image 173, AVI 174 and RAR image 175 based onthe person viewing scene 172 having substrate 182 and object 184 fromPOV2. In contrast to scene image 173 in FIG. 3B, the protruding portionof object 184 is visible, reflecting the rotation of substrate 182 inthe shift from POV1 to POV2. Further reflecting the rotation ofsubstrate 182, the positions of the fiducials 188A-188C in scene image173 are shifted. For example, while fiducial 188A is to the left offiducial 188B in scene image 173 of FIG. 3B, the fiducial 188A is to theright of fiducial 188B in scene image 173 of FIG. 3C. As shown in RARimage 175, AVI 174 is adjusted so that the AVI markers 189A-189C remainsubstantially coincident with shifted fiducials 188A-188C, and AVIconstruct 186 is concomitantly adjusted to appear as a long tubularobject, corresponding to the orientation of object 184 with respect toPOV2. The adjustment of AVI 174, including the orientation of AVIconstruct 186 and the positions of AVI markers 189A-189C, may be basedon comparing the positions of AVI markers 189A-189C with homologousfiducials 188A-188C in the scene image, as the scene image shifts fromPOV1 to POV2, as described hereinabove with reference to FIGS. 2A-2C.

In accordance with an embodiment of the invention, adjustments made inthe AVI construct according to the POV as described above may also beutilized to produce a stereoscopic perception of an AVI construct. TwoRAR apparatuses may be incorporated into a binocular system, each RARapparatus associated with one of the two eyes of the person so thatseparate AVIs may be projected to the two eyes. The difference inposition of the two eyes creates a difference in the positions of thefiducials in the scene image, which results in differences in the AVIconstruct presented to each eye, generating a stereoscopic perception inthe person's mind.

Reference is now made to FIG. 4A, which schematically shows a“split-field RAR apparatus” 200 that is substantially the same as RARapparatus 100, with the addition of an occluder 205 that is operable toblock a portion of the scene light from the scene to create a darkened“occluded field” in the scene image. The occluder is typically mountedfurther away from the eye of the person compared to optics 110 thatdirects the AVI to the person or the camera, so that the occluder blocksonly the scene light.

Reference is now made to FIG. 4B, which schematically shows a“split-field RAR image” 93 that is substantially the same as RAR image75 of FIG. 2A, with the addition of an occluded field 94 and an AVIconstruct 96, also referred to as a “replacement AVI construct”, whichoverlays the occluded field or a portion thereof. By way of example, asshown in FIG. 4B, replacement AVI construct 96 may be a text fielddescribing an object shown in the RAR image. Alternatively, replacementAVI construct 96 may be a video stream. In a surgical application, thevideo stream may, by way of example, be from a camera that is a part ofa minimally invasive surgery tool. Alternatively, replacement AVIconstruct 96 may be an image or a video stream of the scene from adifferent POV or a different scene. The split-field RAR image mayinclude multiple replacement constructs. Because the scene light isblocked in the occluded field, the replacement AVI construct may beviewed by the person without interference from the scene image. Incertain embodiments of the invention, AVI makers may be excluded fromthe portion of the AVI overlaying occluded field 94, to match the lackof fiducials in occluded field 94.

The size and location of the occluded field may be adjustable. Variousmethods and components for adjusting the size and location of theoccluded field are known in the art. By way of example, the occluder mayinclude a movable screen, with the location and size of the movablescreen within the path of the scene light determining the location andsize of the occluded field. The size of the movable screen may beadjustable, or the screen may be replaceable so that a screen of thedesired size may be attached to the occluder.

Reference is now made to FIG. 5, showing an illustration of a particularexample of a scene 372, a scene image 373, an AVI 374, and a RAR image375, in a surgical setting, in accordance with an embodiment of theinvention. In this example, scene 372 is a scene of a surgical site inwhich an opening 362 has been made in the lower torso of a patient 360,which partially reveals a top portion of a kidney 364. Scene image 373may be the surgical site as seen by a surgeon (not shown in FIG. 5). AVI374 includes an image of an AVI construct 386 that is a representationof kidney 364. In RAR image 375, AVI construct 386 is coincident withthe location and orientation of kidney 364 in scene image 372 such thatthe surgeon is able to view “through” the patient to see construct 386together with the surrounding portions of patient 360. The AVI mayinclude other AVI constructs corresponding to other objects such assurgery tools, including implants, catheters, clamps, pads, and thelike, which may be visible or obscured in the scene. In certainembodiments of the invention, the object may not be visible at all tothe surgeon, for example, when the object is situated in a location thatis not exposed by the incision, or when the incision is small, forexample in an endoscopic procedure. By way of example, the object may bea pin to be inserted into a vertebra, and the surgeon inserts the pinvia a small incision without exposing the bone. In such an example, thevertebra itself as well as the pin may be represented in the AVI modeland be part of one or more AVI constructs. Additionally oralternatively, a vector indicating the direction from which the pinshould be inserted and optionally a vector indicating the direction ofthe pin insertion tool may be part of the AVI model and AVIconstruct(s).

In the following detailed description, with reference to FIGS. 6A-6B and7A-7C, examples of a RAR apparatus in accordance with an embodiment ofthe invention, with exemplary optics having one or more mirrors fordirecting scene light from a scene and AVI light projected from aprojector, are discussed. The exemplary optics may include additionaloptical elements, such as lenses, prisms or additional mirrors asneeded, and/or to provide additional functionalities of the optics, suchas magnification and the like. The mirrors in the optics may be made oftransparent glass, plastic sheet, or other materials known in the art,and may have one or more optical coatings, as known in the art, toprovide desired optical transmission and reflection properties. Themirrors may have various shapes and configurations, as known in the art.By way of example, the mirrors may be thin or thick, large or small, ora surface of a beam splitting cube. Particular depiction of the mirrorsin FIGS. 6A-6B and 7A-7C is a schematic drawing that does not excludeany particular embodiment of the invention. The mirrors of the optics,as described with reference to FIGS. 6A-6B and 7A-7C include anydevices, elements, or combinations of elements, having a variety ofshapes that may be used in the art as a mirror.

Reference is now made to FIG. 6A showing a schematic top view of a RARapparatus 400 comprising optics 410, a camera 420, a projector 430 and acontroller 440, in accordance with an embodiment of the invention. Scenelight arriving from a scene is schematically represented by a blockarrow 472. Controller 440 renders an AVI (not shown) in projector 430.Projector 430 projects AVI light, which is schematically represented bya block arrow 474. Optics 410, comprising a mirror 412, receives scenelight 472 and AVI light 474 and directs a portion of AVI light 474 and aportion of scene light 472 to person's 60 eye to generate a PRAR image(not shown) on the person's retina. PRAR light comprising the portion ofAVI light 474 and the portion of scene light 472 directed by optics 410to person's 60 eye is schematically represented by a block arrow labeled476. Additionally, optics 410 directs a portion of AVI light 474 and aportion of scene light 472 to camera 420 to generate a CRAR image (notshown) on the camera's photosensor (not shown). CRAR light comprisingportions of AVI light 474 and scene light 472 directed by optics 410 tothe camera is schematically represented by a block arrow labeled 477. Incertain embodiments of the invention, as shown in FIG. 6A, mirror 412 issituated between the scene and the person and also between the projectorand the camera. It will be appreciated that the various elements of RARapparatus 400 may be situated in various configurations, and thedrawings are not intended to limit the physical arrangement of theelements, For example, the locations of projector 430 and camera 410 maybe reversed so that projector is on the left side of the eye and thecamera is on the right side. Alternatively, the projector may be abovethe eye and the camera may be below the eye. Uses of additional opticalelements to accommodate particular physical arrangements of elements areknown in the art.

According to an embodiment of the invention, mirror 412 may be partiallyreflecting and partially transmitting. Mirror 412 may have one surfacethat is coated with an anti-reflective coating to be mostlynon-reflective, with the other side being coated as desired with layersof materials to be partially reflecting and partially transmitting, asknown in the art. In the discussion below the non-reflective surfacewill be neglected as it does not influence the optical performance ofoptics 410. Alternatively, mirror 412 may be a thin sheet of material,where the reflections from both surfaces have negligible lateral shift.Alternatively, mirror 412 may be a beam splitting cube, with the opticalcoating of interest embedded inside the cube and the other opticalsurfaces of the cube are coated with anti-reflective coatings. Mirror412 by way of example may have a reflection/transmission (“R/T”) ratioof about 50%/50%. Additionally, R/T ratio may be wide-band, beingsubstantially the same across the range of relevant wavelengths, forexample in the visible range or from infrared to ultravioletwavelengths.

In an alternative embodiment of the invention, partially reflectingmirror 412 may comprise a controlled mirror that can change itsreflection and transmission properties over time. Such a mirror, by wayof example, may be a liquid crystal, electro-optics or magneto-opticscell. Such a mirror may be configured to, in alternating intervals, havemore or less transmission or reflection to scene light 472 and AVI light474 projected by projector 430. When the controlled mirror is adjustedto be mostly transparent, substantially all of scene light 472 istransmitted to the person and substantially all of the AVI light 474 istransmitted to the camera. When the controlled mirror is adjusted to bemostly reflective, substantially all of scene light 472 is reflected tothe camera and substantially all of the AVI light 474 is transmitted tothe person. As such, each of PRAR 476 and CRAR 477 comprises pulses ofscene light 472 alternating with pulses of AVI light 474. The R/T ratioin such a configuration may be determined by the proportion of the timethe controlled mirror is reflective compared to the time the controlledmirror is transmissive.

In an alternative embodiment of the invention, partially reflectingmirror 412 may comprise a mirror that has different reflectionproperties on different areas of its surface. For example, mirror 412may have be mostly transmissive on a first area of its surface and bemostly reflective on a second area of its surface. The first and secondareas may be placed into the optical path of scene light 472 and AVIlight 474 in an alternating fashion over time. Such an alternatingplacement may be achieved, for example, with a rotating mirror havingthe first and second areas.

FIG. 6B illustrates directing scene light 472 and AVI light 474 byoptics 410 having mirror 412. Optics 410 generates PRAR light 476 andCRAR light 477 simultaneously. Scene light 472 is split by mirror 412,being partially directed to the camera via reflection and partiallydirected to the person via transmission. AVI light 474 is also split bymirror 412, being partially directed to the person via reflection andpartially directed to the camera via transmission. Thus, CRAR light 477includes the portion of AVI light 474 transmitted through mirror 412 andthe portion of scene light 472 reflected by mirror 412, and PRAR light476 includes the portion of scene light 472 transmitted through mirror412 and the portion of AVI light 474 reflected by mirror 412.

In certain embodiments of the invention, the distance between mirror 412and the eye of the person may be substantially the same as the distancebetween mirror 412 and camera 420. RAR apparatus 400 may be configuredsuch that the front nodal point of the lens of camera 420, as reflectedby the mirror 412, coincides with the front nodal point of the lens inthe eye of the person, as transmitted through the mirror 412. Sucharrangements of the camera and the optics in relation to the person'seye may be advantageous, for example, by preventing or minimizingparallax error between the CRAR image and the PRAR image. In certainembodiments of the invention, mirror 410 may be optically flat, havingno optical power. An optically flat mirror 410 may be advantageous, forexample, by preventing or minimizing magnification differences betweenthe scene image and the AVI in the CRAR image and the PRAR image. Incertain embodiments of the invention, optics 410 may have severaloptical elements having an integrative effect of a flat mirror.

In accordance with an embodiment of the invention, the RAR apparatus maybe configured so that the relative level of luminance between the scenelight and the AVI light is substantially the same in the CRAR light andthe PRAR light.

In certain embodiments of the invention, the R/T ratio of mirror 412 maybe about 50%/50%, resulting in the relative brightness of scene light472 and AVI light 474, as captured by camera 420 and as observed by theperson, being substantially the same. If the R/T ratio departs from50%/50%, the relative brightness of a scene light 472 compared to AVIlight 474 will be different in PRAR light 467 and CRAR light 477 (absentany additional adjustments). In such a case, various luminanceequalization methods may be employed to ensure that the relativebrightness between the scene light 472 and the AVI light 474 issubstantially the same in both PRAR light 476 and CRAR light 477. Allmirrors have some optical loss, and the sum of the reflected light andthe transmitted light is not 100%. However, this optical loss istypically small and will be neglected here for clarity of discussion.The optical loss may be calculated by a person skilled in the art.

Assume by way of example that mirror 412 has a wide-band R/T ratio thatis low reflection/high transmission, as may be advantageous in enablingthe person to perceive the scene in nearly the natural level ofillumination. A first luminance equalization configuration may beemployed, where AVI light 474 is monochromatic, projected substantiallyat a single wavelength or as a combination of discrete monochromaticwavelengths, and mirror 412 comprises one or more optical coatings tunedto enhance reflection for each of the one or more monochromaticwavelengths at which AVI light 474 is projected, so that an R/T ratio,hereinafter referred to as a “monochromatic R/T ratio”, with respect toeach of said monochromatic wavelengths is substantially an inverse ofthe wide-band R/T ratio.

By way of example, mirror 412 may have a wide-band R/T ratio of 20%/80%,with an optical coating tuned to a monochromatic wavelength at whichprojector 430 projects AVI light 474, such that the R/T ratio for thatmonochromatic wavelength is 80%/20%. Given such optics 410, scene light472 is directed to the person at 80% of its original luminance viatransmission (based on the wide-band R/T ratio) and AVI light 474 isdirected to the person at 80% of its original luminance via reflection(based on the monochromatic R/T ratio). At the same time, scene light472 is directed to the camera at 20% of its original luminance viareflection (based on the wide-band R/T ratio) and AVI light 474 isdirected to the camera at 20% of its original luminance via reflection(based on the monochromatic R/T ratio). As such, while the overallluminance of PRAR light 476 is four times the overall luminance of CRARlight 477 (80% directed to the person vs. 20% directed to the camera),the relative luminance between AVI light 474 and scene light 472 in eachof the RAR lights is substantially the same.

Projector 430 projecting AVI light 474 at three monochromaticwavelengths may be advantageous, as it would allow for generating theAVI image in full color, for example, by using a combination ofred-green-blue or other additive primary colors. In accordance with thethree monochromatic wavelengths being projected from projector 430,mirror 412 may comprise optical coatings to provide the monochromaticR/T ratio for each of the three monochromatic wavelengths.

Alternatively or additionally to the first luminance equalizationmethod, a second luminance equalization method may be employed, in whichprojector 430 projects AVI light 474 in a pulsed mode.

Assuming by way of example a mirror 412 having a low R/T ratio, AVIlight 474 is attenuated more than scene light 472 in PRAR light 476because AVI light 474 is directed to the person via reflection whilescene light 472 is directed to the person via transmission. Thisluminance mismatch in PRAR light 476 may be corrected by increasing theluminance at which the projector 430 projects AVI light 474. However,correcting the luminance mismatch in PRAR light 476 in such a wayexacerbates the luminance mismatch in CRAR light 477. In CRAR light 477,due to the same low reflection/high transmission property of mirror 412,it is scene light 472 that is more attenuated than AVI light 474. As aresult, when the luminance mismatch in PRAR light 476 is corrected byincreasing the luminance at which projector 430 projects AVI light 474,an even greater luminance mismatch between AVI light 474 and scene light472 is introduced to CRAR light 477, with the AVI light being muchbrighter than the scene light.

This exacerbated luminance mismatch in CRAR light 477 may then becorrected separately as follows. Projector 430 may be configured toproject AVI light 474 in pulse mode, and the light capturing propertiesof the camera is controlled to change over time. Because AVI light 474is pulsed from projector 430 while scene light 472 is presentedcontinuously, the RAR light has periods of time when scene light 472 ispresented together with AVI light 474, and other periods of time whenscene light 472 is presented without AVI light 474. Thus, the degree towhich AVI light 474 is captured by camera 420 may be tuned independentlyfrom the degree to which scene light 472 is captured by camera 420. Thatis, scene light 472 (which, in CRAR light 477, is much dimmer than AVIlight 474) can be preferentially captured by making camera 420 morereceptive during the phases when scene light 472 is being presented inisolation, and making camera 420 less receptive during the phases whenscene light 472 is presented together with AVI light 474.

As a numerical example of the differential capturing of scene light 472and AVI light 474 by camera 420, assume by way of example that camera420 runs at 100 frames per second (“FPS”), that is, 1 frame per 10milliseconds (msec). Further assume by way of example that projector 430is operated in a pulsed mode such that AVI light 474 is pulsed at 25 FPS(which is equivalent to 1 pulse per 40 msec) with a duty cycle of 25%.Thus, AVI light 474 is projected by the projector at intervals of 10msec that are interspersed by 30 msec rest periods where the AVI lightis not projected. Further, the 25 FPS frame rate of projector 430 andthe 100 FPS frame rate of camera 420 is synchronized so that each 10msec pulse of AVI light 474 coincides with every fourth 10 msec framecaptured by camera 420 (referred to herein as a “coinciding frame”).

Assume that mirror 412 has an R/T ratio of 20%/80%. Assume that theaverage luminance of scene light 472 is 100 units (arbitrary), andprojector 430 projects AVI light 474 at an average luminance of 400units, which is 4× higher than that average luminance of the scenelight. Given this configuration, in PRAR light 476 directed to theperson, scene light 472 is transmitted through mirror 412 at 80 units(or 80% of its original average luminance of 100 units), and AVI light474 is reflected by mirror 412 at the matching luminance of 80 units (or20% of its original average luminance of 400 units). However, therespective luminance of scene light 472 compared to AVI light 474 isskewed in favor of AVI light 474 in CRAR light 477, in which scene light472 is reflected by mirror 412 at 20 units (or 20% of its originalaverage luminance of 100 units), and AVI light 474 is transmittedthrough mirror 412 at 320 units (or 80% of its original averageluminance of 400 units). Thus, in CRAR light 477, the average luminanceof AVI light 474 is about 16 times greater than the average luminance ofscene light 472. Further, due to AVI light 474 being projected onlyduring one of out of every four frames captured by camera 420, theluminance at which AVI light 474 is projected during each coincidingframe, as opposed to the average luminance, is 1600 units (400 units×4).As such, the luminance of AVI light 474 during each coinciding frame is1280 units (80% of 1600 units), and the luminance AVI light during thecoinciding frame is 64 times greater than the luminance of scene light472. To match the average luminance of AVI 474 to the average luminanceof scene light 472, the signal captured by camera 420 from AVI light 474is advantageously appropriately attenuated.

The electrical signal output of camera 420 from AVI light 474 may bepreferentially attenuated by selectively reducing the electrical signaloutput of camera 420 during the coinciding frames. The electrical signaloutput of camera 420 from AVI light 474 may be attenuated by variousmethods such as electrical attenuation, shortening integration times ofthe photo-activated cells of camera 420, computationally reducing thedetected signal by an adjustment factor, inserting an electro-optical ormagneto-optical attenuator in the light path, or changing the reflectionproperties of mirror 412 over time.

The pulsed presentation of the AVI light, together with the inducedmismatch in luminance between the AVI light and the scene light in thesecond luminance equalization method may also be useful for thecontroller to distinguish between the scene image and the AVI, forexample to distinguish between the image of the fiducials and thehomologous AVI markers.

Additionally, differential capturing of the scene light and the AVIlight by the camera may be advantageous, by allowing additional personsviewing the output of camera 420 on a display device (not shown) outsideof the RAR apparatus to control the brightness of the AVI compared tothe scene image as seen on the display device without interfering withthe operation of the RAR apparatus.

In the above first and second luminance equalization methods, theluminance of AVI light 474 was equal to the luminance of scene light 472in PRAR light 476. In certain embodiments of the invention, AVI light474 may be higher or lower in luminance compared to scene light 472 inPRAR light 476, which may be achieved by adjusting the luminance of AVIlight 474 projected by projector 430 accordingly.

Reference is now made to FIG. 7A, showing a schematic top view of a RARapparatus 500, in accordance with an embodiment of the invention. Scenelight arriving from a scene that a person is looking at is schematicallyrepresented by a block arrow 572. RAR apparatus 500 comprises optics510, camera 520, projector 530 and controller 540. Controller 540renders an AVI (not shown) in projector 530. Projector 530 projects AVIlight, schematically represented by a block arrow 574. Optics 510directs a portion of AVI light 574 and a portion of scene light 572 toeye 65 to generate a PRAR image (not shown) on the retina of eye 65. ThePRAR light is schematically represented by a block arrow 576. CRARlight, which includes portions of AVI light 574 and scene light 572directed by optics 510 to the camera to generate a CRAR image (notshown) on a photosensor (not shown) of camera 520, is schematicallyrepresented by a block arrow labeled 577. It will be appreciated thatthe various elements of RAR apparatus 500 may be spatially configured invarious configurations, and the drawing of FIG. 7A is not intended tolimit the physical arrangement of the elements. The use of additionaloptical elements to accommodate particular physical arrangements ofelements is known in the art.

Optics 510 is configured to direct the RAR light to the person andcamera 520 in a serial manner, first directing PRAR light 576 to eye 65,then receiving a reflection of PRAR light 576 from the retina of eye 65and directing the reflection to camera 520 as CRAR light 577.

FIGS. 7B-7C illustrate the paths of the light of the various visualinputs through optics 510. First mirror 512 and second mirror 514 arepartially transmitting and partially reflecting mirrors. Reference isnow made to FIG. 7B, which shows the paths of scene light 572 and AVIlight 574 through optics 510 to form PRAR light 576. For simplicity ofpresentation, some reflections and transmissions created in the opticsthat are not utilized are not shown. Scene light 572 is transmittedthrough mirror 512 towards eye 65. AVI light 574 is transmitted throughmirror 514 and reflected by mirror 512 towards eye 65. The resultingcombination of scene light 572 and AVI light 574 directed to eye 65forms PRAR light 576. Reference is now made to FIG. 7C, which shows thepath of CRAR light 577 through optics 510. CRAR light 577 is areflection of PRAR light 576 from the retina of eye 65, which isreflected by mirror 512 and mirror 514 to be directed to camera 520.Because camera 520 is configured to capture light reflecting from eye65, the controller may further be operable to determine the direction ofgaze of the person by tracking the location of one or more features ofthe eye, by way of example, by tracking the location of the fovea.

In certain embodiments of the invention, optics 410 (as described withreference to FIG. 6A) and optics 510 (as described with reference toFIG. 7A) may be combined into one RAR apparatus (not shown), so that thebenefit of the respective optics may be used as needed for a specificapplication.

According to an embodiment of the invention, a RAR apparatus of anembodiment of the invention may be incorporated in a head-mounteddisplay or a helmet-mounted display, or be worn like a pair of glasses.In certain embodiments of the invention, the RAR apparatus mayincorporate additional optical elements to, for example, createmagnification or reduction in the scene viewed by the person or fold theoptical path in a desired way. Alternatively, the RAR apparatus may beincorporated in an optical instrument that provides magnification orreduction of the scene viewed by the person, including but not limitedto a microscope, a telescope, a binocular, or a pair of glasses. The RARapparatus may be integrated within another device or system, or may beretrofittable.

In certain embodiments of the invention, the RAR apparatus may beincorporated into a monocular system. Alternatively, two RAR apparatusesmay be incorporated into a binocular system. In the binocular system,the two RAR apparatuses may independently project AVI light to each eyeto generate the respective AVIs. Because the positions of the fiducialsis slightly different in the respective scene image of each eye, each ofthe two RAR apparatuses renders a different AVI construct that isprojected to each eye to generate a stereoscopic perception in theperson's mind. In certain embodiments of the invention, one controllermay be utilized for the two RAR apparatuses, rendering two AVIs havingdifferent AVI constructs for projection by the two projectors.

Reference is now made to FIG. 8A, which is a flowchart of a method forproviding a person with an AR responsive to a scene that the personviews, the method comprising: rendering an AVI that has AVI markershomologous to fiducials in the scene (1020); projecting the AVI (1025);directing the AVI and an image of the scene to the person's eye and acamera (1030); capturing the AVI and the image of the scene (1035);comparing positions of AVI markers in the captured AVI with positions ofhomologous fiducials in the captured scene image (1040); rendering anadjusted AVI in which the AVI markers are substantially coincident withthe homologous fiducials in the captured scene image (1050); projectingthe adjusted AVI (1055); and directing the adjusted AVI and the image ofthe scene to the person's eye and the camera. (1060).

In certain embodiments of the invention, the method may be a repeatingprocess. Thus, following the completion of block 1060, where theadjusted AVI and scene image are directed to the person's eye andcamera, the method may repeat from block 1035, to capture the (adjusted)AVI and the scene image. Each repeated cycle of the method may bereferred to as a “viewing cycle”. The viewing cycles may continue for a“session” of multiple viewing cycles for a predetermined time or untilstopped by the person's command. This repeating process may allow forthe alignment between the scene image and the AVI to be continuouslyimproved in an iterative process, in which the AVI is repeatedlyadjusted until the degree of alignment between the AVI markers and theimages of the fiducials reach or go below a predefined threshold.Additionally or alternatively, the repeating process may allow for theAVI to be continuously responsive to changes in the scene image, whichmay be caused by changes in the person's head location or the movementof objects in the scene.

According to an embodiment of the invention, the AVI rendered in block1020 may be an initial AVI. In certain embodiments of the invention, theinitial AVI may be rendered based on one or more estimation methods thatdo not require comparing the positions of the AVI markers with thefiducials in the scene. The one or more estimation methods mayoptionally be based on one or a combination of: an assumed position ofthe person and the object, data from external sensors configured todetect the position of the person's head, a user-provided estimated headposition, or the like. Alternatively or in addition, the initial AVI maybe rendered based on the positions of the fiducials in an initiallycaptured scene image. In certain embodiments of the invention, theinitial AVI may comprise only the AVI markers, so that the AVI constructis first presented in the adjusted AVI based on the positionalcomparison of the initial AVI markers with the image of the homologousfiducials.

With reference to FIG. 8B, in certain embodiments of the invention, themethod may be preceded by the creating of a new AVI model, which mayinclude creating a new construct model and/or a new marker model (1010),where the AVI rendered in block 1020 is based on the new AVI modelcreated in block 1010. The new AVI model may be a reconstruction fromone or more images captured from the scene. The images may be capturedthrough optical imaging, or imaging by means of an alternative imagingmodality, such as but not limited to X-ray, MRI, ultrasound and thelike. The imaging may be a tomographic imaging, such as but not limitedto a CT scan, a PET scan, a tomographic MRI, an ultrasound tomography orthe like. Where the scene and the objects within it are unstable orsubject to movement and change, it may be advantageous to precede atleast a portion of the viewing cycles with creating a new constructmodel and/or marker model.

In a repeating process, after directing the adjusted AVI and the sceneimage to the person's eye and camera (1060), the method may engage block1035 to capture the (adjusted) AVI and the scene image or,alternatively, the method may engage block 1010 to create a newconstruct model and/or a new construct model. The method may engageblock 1010 instead of block 1035 in every viewing cycle or in a subsetof the viewing cycles, for example, every ten viewing cycles, everyother viewing cycle or the like. Alternatively, step 1010 may be engagedinstead of block 1035 in individual viewing cycles according to thecapabilities of the viewing equipment or as desired and directed by theperson.

In certain embodiments of the invention, the method may be implementedin an RAR apparatus according to an embodiment of the invention.

Certain embodiments of the invention may have applications in surgicalprocedures. Other applications may also be managed similarly, with somemodifications that depend on the particular application. Some examplesof applications, not by any means an exhaustive list, are given below.

The object may be a building or portions thereof. The construct model(s)visualized in the AVI may be derived from construction plans. Theconstruct model may include locations of people inside the building, forexample, guests, personnel or emergency responders such as firefighters.The corners of the building itself and corners of windows and doors, orthe contour lines of the building, may be used as fiducials. Thelocation of the people may be derived from GPS or mobile phone signals.If the object is missing defined points that may serve as fiducials (forexample in a disaster-struck area), a laser can generate several beamsthat serve as artificial indicators to create the fiducials.

As applied in ship navigation, the ship, underwater terrain features andsurrounding underwater items may be objects used to generate constructmodels. The construct models may be generated from underwater scans,such as Sonar, Lidar, map features, etc. Items that are consistentlyviewed by the person, for example, items inside the control bridge or onparts of the ship visible from the bridge, may serve as fiducials. Thenavigator's natural view may be augmented with AVI constructsrepresenting, for example, the belly of the ship, underwater obstaclesand wildlife, vector representation of the ship's motion (or the motionof any other item, for example, another ship) and the like. A protectedvolume around the ship may be one of the AVI constructs, which may bedependent on the speed vector of the ship and time to impact, and anypenetration of the protected volume may be configured to be accompaniedby a visual alarm embedded in the AVI or by an audible alarm.

In an exemplary educational application, the AVI may include AVIconstructs representing internal parts of an engine—stationary or inmotion during operation.

In the assembly and repair of complex structures, the operator may viewan AVI comprising AVI constructs representing parts that need to beinstalled, removed or moved, how to hold such parts, what screws toinstall or remove, where to connect electrical probes, etc.

For excavation operators, the AVI may include AVI constructsrepresenting underground cables and pipes while digging next to suchelements. The construct model(s) may be generated using, for example,electromagnetic sensors, the coupling of transmission equipment to thecable or pipe, acoustic transmitters and receivers, radiolocation,ground penetrating radar, sonar, etc. Artificial indicators for creatingfiducials may be, for example, acoustic or electromagnetic transmitters,blocks of heavy metal, etc. that can be sensed by one or more of thevarious sensors.

Air traffic controllers may be provided with an AVI comprising AVIconstructs representing, for example, airplanes, vehicles, humans, etc.in an airport, including those objects located behind visual obstaclesor obscured in a foggy atmosphere. Each airplane and auto may have addedfeatures such as a motion vectors and/or warning signs for convergingvectors. The construct model(s) may be generated using information fromone or more of the following: radar, sonar, millimeter wave radar,thermal imaging cameras, IR cameras, GPS reading from units installed onairplanes and autos, etc. The fiducials may be, for example, objectsmounted inside the tower or structures in the airfield visible from thetower(s).

In the description and claims of the present application, each of theverbs, “comprise” “include” and “have”, and conjugates thereof, are usedto indicate that the object or objects of the verb are not necessarily acomplete listing of components, elements or parts of the subject orsubjects of the verb.

Descriptions of embodiments of the invention in the present applicationare provided by way of example and are not intended to limit the scopeof the invention. The described embodiments comprise different features,not all of which are required in all embodiments of the invention. Someembodiments utilize only some of the features or possible combinationsof the features. Variations of embodiments of the invention that aredescribed, and embodiments of the invention comprising differentcombinations of features noted in the described embodiments, will occurto persons of the art. The scope of the invention is limited only by theclaims.

1. (canceled)
 2. The apparatus of claim 1, wherein the apparatus issubstantially free of one or more sources of relative dimensionalvariance between the first combined image and the second combined imageselected from the group consisting of: parallax error due to thepositioning of the retina of the person and the photosensor,differential chromatic magnification and optical aberrations.
 3. Theapparatus of claim 22 wherein the optics comprises a mirror that ispartially transmitting and partially reflecting.
 4. The apparatus ofclaim 22, wherein the AVI includes an image representing an objectwithin the scene, the image being rendered from a computer code encodinga three-dimensional representation of the object.
 5. The apparatus ofclaim 4, wherein the computer code is a reconstruction based on one ormore captured images of the object.
 6. The apparatus of claim 22,wherein the homologous features in the scene are native to the scene. 7.The apparatus of claim 22, wherein the homologous natures comprise oneor more artificial indicators placed on the scene.
 8. The apparatus ofclaim 22, wherein the optics is configured to make the relativeluminance between the image of the scene and the AVI directed to theperson substantially the same as relative luminance between the image ofthe scene and the AVI directed to the photosensor.
 9. The apparatus ofclaim 22, wherein the optics is configured to have light in the combinedimage reflecting from the retina be directed to the photosensor.
 10. Theapparatus of claim 22 configured to be incorporated into a head-mountedapparatus.
 11. The apparatus of claim 22 configured to be combined withan optical instrument that provides magnification or reduction of thescene.
 12. A method for providing a person with an augmented reality,the method comprising: rendering an augmenting visual image (AVI)comprising location markers corresponding to selected homologousfeatures in a scene; projecting the AVI, wherein the AVI is notprojected to the scene; directing the AVI and an image of the scene tothe person's retina and a photosensor; capturing the AVI and the imageof the scene; comparing positions of the homologous features in theimage of the scene with positions of the corresponding location markersin the AVI; rendering an adjusted AVI in which the location markers aresubstantially coincident with the corresponding homologous features inthe capture scene image; and projecting the adjusted AVI, wherein theAVI is not projected to the scene; and directing the adjusted AVI andthe image of the scene to the person's retina and the photosensor. 13.The method of claim 12, wherein following directing the adjusted AVI andthe image of the scene, the method is repeated from capturing the AVIand the image of the scene.
 14. The method of claim 12, wherein the AVIincludes an image representing an object within the scene, the imagebeing rendered from a computer code encoding a three-dimensionalrepresentation of the object.
 15. The method of claim 14, preceded bycreating a new computer code encoding the three-dimensionalrepresentation of the object.
 16. The method of claim 14, wherein thecomputer code encoding the three-dimensional representation of theobject is a reconstruction based on one or more captured images of theobject.
 17. The method of claim 15, wherein, following directing theadjusted AVI and the image of the scene, the method is repeated fromcreating the new computer code encoding the three-dimensionalrepresentation of the object.
 18. The method of claim 12, wherein thehomologous features in the scene are native to the scene.
 19. The methodof claim 12, wherein the homologous features in the scene comprise oneor more artificial markers placed on the scene.
 20. An apparatus forproviding a person with an augmented reality responsive to a scene thatthe person views, the apparatus comprising: a photosensor; a lightprojector that projects light that generates an augmenting virtual image(AVI) that is not projected onto the scene; optics comprising apartially reflecting flat mirror situated between a scene and theperson's retina and also between the projector and the photosensor, thepartially reflecting flat mirror being configured to direct portions ofthe light from the scene to the person's retina and to the photosensor,and to direct portions of the light from the projector to the person'sretina and to the photosensor.
 21. The apparatus of claim 20 furthercomprising an occluder situated between the scene and the optics.
 22. Anapparatus for providing a person with an augmented reality responsive toa scene that the person views, the apparatus comprising: a photosensorthat images light incident on the photosensor; alight projector; acontroller that controls the projector to project light that generatesan augmenting virtual image (AVI) that is not projected onto the sceneand having markers corresponding to homologous features in the scene;optics that directs light from the scene and light projected by theprojector to form a first combined image on the person's retinacomprising an image of the scene overlaid by the AVI, and a secondcombined image comprising an image of the scene overlaid by the AVI onthe photosensor; wherein the controller configures the AVI responsive tothe second combined image so that the markers in the first combinedimage are substantially coincident with the corresponding homologousfeatures of the scene comprised in the first combined image.